In many imaging applications, users may wish to access a random viewing angle of a captured scene in high resolution, even when this specifically queried imagery is not among the set of acquired sample images. In theory, 3D reconstruction based rendering can be applied to generate such an (artificial) image. However, accurate camera calibration over large scale photo collections is needed and is highly complex in nature. Image stitching based approaches, such as panorama imaging, can also be applied. However, such schemes are unable to provide free view interpolation or resolution enhancement.
Image-based rendering (IBR) is a technique that renders novel views of the scene given a set of images, and has long been studied in computer graphics and computer vision. With the rapid progress and extensive deployment of mobile devices in recent years, there is strong consumer interest in developing light weight algorithms capable of high quality free-view interpolation with zoom-in/out effects given limited uncalibrated views.
Two widely used approaches for IBR are 3D points cloud reconstruction and image stitching, respectively. The resolution of the synthesized view of 3D reconstruction based IBR depends highly on the accuracy of the recovered points cloud. In this case, a large number of reference images from varying views are required for calibration. For image stitching based IBR, even though strict controls over camera positions are usually required, serious artifacts may still occur at the seam because of the ignorance of depth and view disparity during image registration, and due to the non-redundant information in the overlapping area of adjacent frames. Both approaches suffer from several unresolved issues when high resolution free view interpolation is desired. This is particularly true for mobile devices when limited computation resources are available and the acquisition of reference images cannot be properly controlled. In general, mobile users prefer to render a novel view of scene within a short time, based on a few available views, either taken by users themselves or acquired from the Internet.
To determine where a point appears in a specific view given its positions in a set of images, one straightforward and common way is to register all camera poses, reconstruct the 3D scene, and then project the 3D point onto the desired view. For example, where techniques for registration and rendering large scale photo collections are involved, the method may include interactively browsing and exploring large, unstructured collections of photographs. More robust 3D Six Degrees of Freedom (6DOF) registration and scene specific controls can be achieved by related techniques. The precise calibration of full 3D camera poses for all reference images is achieved by leveraging structure-from-motion algorithms with high time complexity, in which multi-core GPUs have been proposed. However, the acquisition and calibration of large scale photo collections is still a huge challenge for users with limited resource and real time needs.
Image stitching based IBR, on the other hand, usually has low time complexity, and is available even in consumer cameras, for example, to obtain a panorama shot. However, control over images is usually required for panorama generation. For standard panoramas, for example, all images are assumed to maintain the same camera center and to compose a wide field of view. On the other hand, for multi-perspective panoramas, reference images are required to correspond to a specific orbit, although the final mosaic combines views from different directions. To minimize the artifacts caused primarily by the disparities between images, dense scene depth can be estimated and structure-from-motion algorithms have also been employed for registration and determination of projection surfaces. For both types of panoramas, proper segments are cropped, registered and stitched. Unfortunately, high resolution with free view has heretofore been virtually impossible to achieve, since the redundant information in the overlapping areas of adjacent views are typically not used for anything other than seam smoothing.
When limited in planar structure, 2D based high resolution free view interpolation can be regarded as a special case of general Super Resolution (SR) problem in 3D space. Super Resolution has long been studied to generate high resolution image by combining the non-redundant information from multiple low-resolution images. Image registration and SR are often treated as distinct and sequential processes. More recently, a Bayesian approach has been developed to estimate SR by optimizing the marginal probability of the observed low-resolution images directly. Lower memory requirements and sharper results are reported by integration over the point-spread function (PSF) and motion parameters. However, all previous SR methods can only treat slight rotation and translation over reference images. These registration schemes would fail when there is disparity in either depth, pitch, or yaw in reference images.